<html>
<head><meta charset="utf-8"><title>Rust malloc issues with long string in macOS · general · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/index.html">general</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Rust.20malloc.20issues.20with.20long.20string.20in.20macOS.html">Rust malloc issues with long string in macOS</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="210828930"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Rust%20malloc%20issues%20with%20long%20string%20in%20macOS/near/210828930" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Phodal Huang <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Rust.20malloc.20issues.20with.20long.20string.20in.20macOS.html#210828930">(Sep 22 2020 at 05:17)</a>:</h4>
<p>Hello all.  New to zulip, I don't know how to resolve this issue, sure, I come here. In my situation, I had a Rust app and it will pass String pointer to C part. </p>
<p>In the short string, it works perfect. But once it's a long string,  It will cause issues. I had do some research like in <a href="https://notes.iveselov.info/programming/allocation-api-and-allocators">https://notes.iveselov.info/programming/allocation-api-and-allocators</a>, it seems a <strong>RT Allocated</strong> issues. </p>
<p>here is my code: <a href="https://github.com/phodal/rust-malloc-issues-test">https://github.com/phodal/rust-malloc-issues-test</a></p>



<a name="210829042"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Rust%20malloc%20issues%20with%20long%20string%20in%20macOS/near/210829042" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Phodal Huang <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Rust.20malloc.20issues.20with.20long.20string.20in.20macOS.html#210829042">(Sep 22 2020 at 05:20)</a>:</h4>
<p>Sorry,  I'm not good at this chat tool, here is the code</p>
<div class="codehilite" data-code-language="Rust"><pre><span></span><code><span class="k">use</span><span class="w"> </span><span class="n">libc</span>::<span class="n">malloc</span><span class="p">;</span><span class="w"></span>

<span class="k">struct</span> <span class="nc">NormalString</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="n">length</span>: <span class="kt">i32</span><span class="p">,</span><span class="w"></span>
<span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="kt">str</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>

<span class="k">impl</span><span class="w"> </span><span class="n">NormalString</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="kt">str</span>: <span class="nb">String</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Self</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="n">NormalString</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">length</span>: <span class="kt">str</span><span class="p">.</span><span class="n">clone</span><span class="p">().</span><span class="n">len</span><span class="p">()</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">i32</span><span class="p">,</span><span class="w"> </span><span class="kt">str</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w">    </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>

<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">long_str</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"Boolean"</span><span class="p">;</span><span class="w"></span>
<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">utf_string</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">NormalString</span>::<span class="n">new</span><span class="p">(</span><span class="nb">String</span>::<span class="n">from</span><span class="p">(</span><span class="n">long_str</span><span class="p">));</span><span class="w"></span>
<span class="w">    </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">malloc</span><span class="p">(</span><span class="n">utf_string</span><span class="p">.</span><span class="n">length</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="kt">u8</span><span class="p">;</span><span class="w"></span>
<span class="w">        </span><span class="n">println</span><span class="o">!</span><span class="p">(</span><span class="s">"{:?}"</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="n">x</span><span class="p">);</span><span class="w">                  </span><span class="c1">// B</span>
<span class="w">        </span><span class="n">println</span><span class="o">!</span><span class="p">(</span><span class="s">"{:?}"</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="n">x</span><span class="p">.</span><span class="n">offset</span><span class="p">(</span><span class="mi">1</span><span class="p">));</span><span class="w"> </span><span class="c1">// o</span>
<span class="w">        </span><span class="n">println</span><span class="o">!</span><span class="p">(</span><span class="s">"{:?}"</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="n">x</span><span class="p">.</span><span class="n">offset</span><span class="p">(</span><span class="mi">2</span><span class="p">));</span><span class="w"> </span><span class="c1">// o</span>
<span class="w">        </span><span class="n">println</span><span class="o">!</span><span class="p">(</span><span class="s">"{:?}"</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="n">x</span><span class="p">.</span><span class="n">offset</span><span class="p">(</span><span class="mi">3</span><span class="p">));</span><span class="w"> </span><span class="c1">// l</span>
<span class="w">        </span><span class="n">println</span><span class="o">!</span><span class="p">(</span><span class="s">"{:?}"</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="n">x</span><span class="p">.</span><span class="n">offset</span><span class="p">(</span><span class="mi">4</span><span class="p">));</span><span class="w"> </span><span class="c1">// e</span>
<span class="w">    </span><span class="p">}</span><span class="w"></span>

<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">long_str2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"</span><span class="se">\\</span><span class="s">b(AbsoluteTime|Boolean|Byte|ByteCount|ByteOffset|BytePtr|CompTimeValue|ConstLogicalAddress|ConstStrFileNameParam|ConstStringPtr|Duration|Fixed|FixedPtr|Float32|Float32Point|Float64|Float80|Float96|FourCharCode|Fract|FractPtr|Handle|ItemCount|LogicalAddress|OptionBits|OSErr|OSStatus|OSType|OSTypePtr|PhysicalAddress|ProcessSerialNumber|ProcessSerialNumberPtr|ProcHandle|Ptr|ResType|ResTypePtr|ShortFixed|ShortFixedPtr|SignedByte|SInt16|SInt32|SInt64|SInt8|Size|StrFileName|StringHandle|StringPtr|TimeBase|TimeRecord|TimeScale|TimeValue|TimeValue64|UInt16|UInt32|UInt64|UInt8|UniChar|UniCharCount|UniCharCountPtr|UniCharPtr|UnicodeScalarValue|UniversalProcHandle|UniversalProcPtr|UnsignedFixed|UnsignedFixedPtr|UnsignedWide|UTF16Char|UTF32Char|UTF8Char)</span><span class="se">\\</span><span class="s">b"</span><span class="p">;</span><span class="w"></span>
<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">utf_string2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">NormalString</span>::<span class="n">new</span><span class="p">(</span><span class="nb">String</span>::<span class="n">from</span><span class="p">(</span><span class="n">long_str2</span><span class="p">));</span><span class="w"></span>
<span class="w">    </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">malloc</span><span class="p">(</span><span class="n">utf_string2</span><span class="p">.</span><span class="n">length</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="kt">u8</span><span class="p">;</span><span class="w"></span>
<span class="w">        </span><span class="n">println</span><span class="o">!</span><span class="p">(</span><span class="s">"{:?}"</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="n">x</span><span class="p">);</span><span class="w"></span>
<span class="w">    </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>

<p>it will return those in my macOS</p>
<div class="codehilite"><pre><span></span><code>66
0
</code></pre></div>


<p>but it return 0 0 in  the Rust Playgroud: <a href="https://play.rust-lang.org/?version=stable&amp;mode=release&amp;edition=2018&amp;gist=240078be1a848cca4f3a0627882a0aa4">https://play.rust-lang.org/?version=stable&amp;mode=release&amp;edition=2018&amp;gist=240078be1a848cca4f3a0627882a0aa4</a></p>
<p>I just try to figure out how to resolve this, and pass String pointer to C part.  But don't have a good point. Does any one have some ideas about to do it?</p>



<a name="210829059"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Rust%20malloc%20issues%20with%20long%20string%20in%20macOS/near/210829059" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lokathor <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Rust.20malloc.20issues.20with.20long.20string.20in.20macOS.html#210829059">(Sep 22 2020 at 05:21)</a>:</h4>
<p>It looks like you're using <code>malloc</code> and then not actually writing anything to the pointer. When you first get the pointer back it points to uninitialized memory, so reading the pointer will be UB.</p>



<a name="210829249"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Rust%20malloc%20issues%20with%20long%20string%20in%20macOS/near/210829249" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Phodal Huang <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Rust.20malloc.20issues.20with.20long.20string.20in.20macOS.html#210829249">(Sep 22 2020 at 05:26)</a>:</h4>
<p>It seems so, but I still wonder why in Rust playground, will have different results. According the article in the mention, in different OS will had different RT allocated:</p>
<table>
<thead>
<tr>
<th>OS</th>
<th>RT Allocated, bytes</th>
<th>Remaining in use, bytes</th>
</tr>
</thead>
<tbody>
<tr>
<td>macOS</td>
<td>241</td>
<td>237</td>
</tr>
<tr>
<td>Windows</td>
<td>441</td>
<td>173</td>
</tr>
<tr>
<td>Linux</td>
<td>177</td>
<td>173</td>
</tr>
</tbody>
</table>
<p>In my system, it's 241. Once strings length &gt; 241, will return 0. But in the playground, it's 0 even length great then 700</p>



<a name="210829321"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Rust%20malloc%20issues%20with%20long%20string%20in%20macOS/near/210829321" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lokathor <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Rust.20malloc.20issues.20with.20long.20string.20in.20macOS.html#210829321">(Sep 22 2020 at 05:28)</a>:</h4>
<p>Well, once UB happens you can't reliably say what will occur. That's why you shouldn't ever perform UB, after all.</p>
<p>Honestly, i'm surprised that there's any consistency to begin with.</p>



<a name="210829413"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Rust%20malloc%20issues%20with%20long%20string%20in%20macOS/near/210829413" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Phodal Huang <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Rust.20malloc.20issues.20with.20long.20string.20in.20macOS.html#210829413">(Sep 22 2020 at 05:31)</a>:</h4>
<p>So, in my situation,  the better idea is rewrite the NormalString part in C, and pass parameters to it?</p>



<a name="210829570"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Rust%20malloc%20issues%20with%20long%20string%20in%20macOS/near/210829570" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Rust.20malloc.20issues.20with.20long.20string.20in.20macOS.html#210829570">(Sep 22 2020 at 05:36)</a>:</h4>
<p>it's UB in C too</p>



<a name="210829614"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Rust%20malloc%20issues%20with%20long%20string%20in%20macOS/near/210829614" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Rust.20malloc.20issues.20with.20long.20string.20in.20macOS.html#210829614">(Sep 22 2020 at 05:36)</a>:</h4>
<p>why are you reading from freshly allocated memory?</p>



<a name="210829767"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Rust%20malloc%20issues%20with%20long%20string%20in%20macOS/near/210829767" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Phodal Huang <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Rust.20malloc.20issues.20with.20long.20string.20in.20macOS.html#210829767">(Sep 22 2020 at 05:40)</a>:</h4>
<p>I try to rewrite VSCode-oniguruma with Rust <a href="https://github.com/microsoft/vscode-oniguruma/blob/2381c655be380c16b7045106fa2acb9610016922/src/index.ts#L250">https://github.com/microsoft/vscode-oniguruma/blob/2381c655be380c16b7045106fa2acb9610016922/src/index.ts#L250</a> . It read it works. And in my cases,  it works in short strings. I think it don't make senses, but it works..., So I try to figure out why it happens in Rust too ?</p>



<a name="210830199"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Rust%20malloc%20issues%20with%20long%20string%20in%20macOS/near/210830199" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Rust.20malloc.20issues.20with.20long.20string.20in.20macOS.html#210830199">(Sep 22 2020 at 05:51)</a>:</h4>
<p>In your example, you seem to be indicating that reading bytes out of <code>x</code>, which yields <code>B,o,o,l,e</code> (which are in fact the characters of a completely separate variable <code>long_str</code>), is the desired behavior. If so, why don't you just take the address of <code>long_str</code>'s data instead, as a <code>*mut u8</code>? That can be done safely as long as you don't change the length of the string</p>



<a name="210830340"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Rust%20malloc%20issues%20with%20long%20string%20in%20macOS/near/210830340" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Rust.20malloc.20issues.20with.20long.20string.20in.20macOS.html#210830340">(Sep 22 2020 at 05:55)</a>:</h4>
<p>If you want to create a C string from a rust <code>&amp;str</code> or <code>String</code>, you should use <a href="https://doc.rust-lang.org/std/ffi/struct.CString.html">https://doc.rust-lang.org/std/ffi/struct.CString.html</a></p>



<a name="210830646"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Rust%20malloc%20issues%20with%20long%20string%20in%20macOS/near/210830646" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Phodal Huang <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Rust.20malloc.20issues.20with.20long.20string.20in.20macOS.html#210830646">(Sep 22 2020 at 06:01)</a>:</h4>
<p>Thanks <span class="user-mention" data-user-id="271719">@Mario Carneiro</span> , it's awesome.</p>



<a name="210853183"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Rust%20malloc%20issues%20with%20long%20string%20in%20macOS/near/210853183" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Daniel Henry-Mantilla <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Rust.20malloc.20issues.20with.20long.20string.20in.20macOS.html#210853183">(Sep 22 2020 at 10:43)</a>:</h4>
<p><span class="user-mention silent" data-user-id="224471">Lokathor</span> <a href="#narrow/stream/122651-general/topic/Rust.20malloc.20issues.20with.20long.20string.20in.20macOS/near/210829321">said</a>:</p>
<blockquote>
<p>Well, once UB happens you can't reliably say what will occur. That's why you shouldn't ever perform UB, after all.</p>
<p>Honestly, i'm surprised that there's any consistency to begin with.</p>
</blockquote>
<p>Funnily enough, the useless <code>.clone()</code> that is performed beforehand, coupled with the fact the allocator was also using <code>malloc</code> as its backend, is what lead to this "fortunate" random-bytes-from-allocator-do-actually-match-those-wanted. It seems two wrongs can make a right, in UB-land <span aria-label="big smile" class="emoji emoji-1f604" role="img" title="big smile">:big_smile:</span></p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>